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Abstract  and  Summary 

The  rapid  proliferation  of  cell  phones,  new  social  network  infrastructure  such  as  Twitter  and  Facebook,  and  a  new 
generation  of  “digital  natives”  who  routinely  collaborate  in  a  distributed  ad  hoc  manner,  provide  a  major  new  source 
of  input  for  situation  assessment  systems,  namely  humans  acting  as  “soft”  sensors.  Human  reports  and 
observations  augment  traditional  “hard”  sensor  systems  such  as  radar,  electro-optic  and  acoustic  sensors.  This 
project  focused  on  understanding  how  to  understand  and  model  the  role  of  human  observers  for  the  ultimate  fusion 
of  hard  and  soft  data. 

This  project  has  provided  the  basis  for  establishing  a  new  discipline  of  hard  and  soft  information  fusion.  At  the 
start  of  this  project,  only  a  limited  amount  of  research  had  been  conducted  in  this  area.  The  concepts  of 
“participatory  sensing”,  “human-centered  information  fusion”  and  “hard  and  soft  information  fusion”  were  just 
emerging  in  the  research  and  application  community.  In  part  as  a  result  of  this  project  a  research  community  is 
growing  and  regularly  reporting  their  research  results  at  conferences  such  as;  (1)  the  annual  International  Conference 
on  Information  Fusion  (ISIF)  which  has  hosted  special  tracks  on  hard/soft  fusion  for  the  past  2  years,  (2)  the  annual 
national  symposium  on  sensor  and  data  fusion  (NSSDF),  and  (3)  the  annual  Beaver  Hollow  Workshop  hosted  by  the 
Center  for  Multisource  Information  Fusion.  These  concepts  are  also  being  utilized  for  studies  such  as  an  evaluation 
of  operations  at  the  Joint  IED  Defeat  Organization  (JIEDDO),  evaluation  of  technologies  for  the  National  Geospatial 
Intelligence  Agency  (NGA),  and  other  applications. 

During  this  project  a  number  of  accomplishments  are  noted; 

(1)  Development  of  the  concept  of  human-centered  information  fusion  (which  encompasses  the  use  of  human 
observers  and  human  in  the  loop  fusion  processing)  (see  [1],  [2],  and  [3]) 

(2)  Conduct  of  an  extensive  literature  review  of  participatory  sensing  [4], 

(3)  Development  of  a  theoretical  framework  for  fusion  of  hard  and  soft  data  (  [1],  [5],  [6],  [17],  [27],  [28]), 

(4)  Conduct  of  an  extensive  literature  review  of  modeling  techniques  for  representing  human  behavior  [7], 

(5)  Identification  of  algorithms  for  fusion  of  hard  and  soft  data  ([8],  [9],  [10]), 

(6)  Development  of  an  Extreme  Events  Laboratory  to  conduct  human  in  the  loop  experiments  ([11],  [12]), 

(7)  Development  of  methods  for  test  and  evaluation  of  hard  and  soft  fusion  systems  ([13],  [14],  [30]), 

(8)  Design  and  conduct  of  a  human  in  the  loop  experiment  focused  on  knowledge  elicitation  and  direction  of 
attention  for  a  human  observer  [15],  and 

(9)  Initial  demonstrations  performed  to  illustrate  the  use  of  Twitter  for  data  collection  and  reporting  ([16],  [26], 

[31]). 

Technology  transfer  occurred  via  hosted  workshops,  special  sessions  in  International  Information  Fusion 
conferences,  meetings  with  government  and  industry  personnel  and  technical  reports  for  the  Joint  IED  Defeat 
Organization  (JIEDDO)  and  the  National  Geospatial  Intelligence  Agency. 


1.0  Background 


This  report  provides  a  summary  of  the  activities,  progress  and  approach  for  the  Modeling  and  Mapping  of  Human 
Source  Data  project  (contract  no:  W  91  INF-07-1-0168)  for  the  U.  S.  Department  of  the  Army.  The  project  was 
conducted  by  a  research  team  led  by  the  Pennsylvania  State  University  College  of  Information  Sciences  and 
Technology  (1ST),  with  support  from  Dr.  James  Llinas  of  Multisource.  This  three  phase  project  was  aimed  at 
developing  concepts,  algorithms  and  techniques  to  allow  incorporation  of  human  source  data  (e.g.,  human 
observations,  comments  and  inferences)  with  traditional  sensor  data  for  improved  situational  awareness  and 
decision-making.  The  premise  is  that  the  rapid  evolution  of  web  technology  coupled  with  wide -band  ubiquitous 
communications  and  personal  devices  such  as  cell  phones,  personal  data  assistants  (PDAs)  and  laptop  computers 
provides  the  opportunity  to  augment  traditional  sensor  systems  with  humans  acting  as  “soft”  sensors  in  network 
centric  operations.  For  military  applications  this  provides  a  basis  for  “every  soldier  (acting  as)  a  sensor”.  For 
operations  such  as  humanitarian  relief  and  disaster  recovery,  in  which  military  personnel  must  operate  in 
conjunction  with  civilians  and  non-government  organizations  (NGOs),  this  concept  becomes  even  more  important  to 
enable  dynamic  ad  hoc  communities  of  observers  to  assist  in  characterizing,  analyzing  and  understanding  evolving 
situations. 

The  objectives  of  this  project  were  to:  (1)  develop  a  methodology  by  which  a  complete  and  robust  approach  to 
“source  modeling”  or  “source  characterization”  of  the  human-based  inputs  can  be  carried  out,  (2)  characterize  a 
design  approach  to  a  data  fusion  process  that  accommodates  the  particular  aspects  of  such  human-based  report 
inputs,  (3)  implement  prototypes  for  data  acquisition  and  knowledge  extraction  from  human  observers  and 
techniques  for  fusing  soft  and  hard  sensor  data,  and  (4)  conduct  limited  human-in-the-loop  experiments.  One  focus 
of  the  research  was  on  the  translation  of  human  factors  and  uncertainties  into  probabilities  or  appropriate  uncertainty 
representations  that  are  judged  necessary  to  deal  with  the  idiosyncrasies  of  human  observation  under  stress,  etc.  The 
overarching  challenge  is  to  develop  an  approach  by  which  the  errors  through  the  observational  and  preprocessing 
chains  can  be  characterized  and  then  employed  in  the  design  of  a  fusion  approach. 

2.0  Summary  of  Results 

Dining  this  project  a  number  of  accomplishments  are  noted; 

(1)  Development  of  the  concept  of  human-centered  information  fusion  (which  encompasses  the  use  of  human 
observers  and  human  in  the  loop  fusion  processing)  (see  [1],  [2],  and  [3]) 

(2)  Conduct  of  an  extensive  literature  review  of  participatory  sensing  [4], 

(3)  Development  of  a  theoretical  framework  for  fusion  of  hard  and  soft  data  (  [1],  [5],  [6]), 

(4)  Conduct  of  an  extensive  literature  review  of  modeling  techniques  for  representing  human  behavior  [7], 

(5)  Identification  of  algorithms  for  fusion  of  hard  and  soft  data  ([8],  [9],  [10],  [17],  [27],  [28]), 

(6)  Development  of  an  Extreme  Events  Laboratory  to  conduct  human  in  the  loop  experiments  ([11],  [12]), 

(7)  Development  of  methods  for  test  and  evaluation  of  hard  and  soft  fusion  systems  ([13],  [14],  [30]), 

(8)  Design  and  conduct  of  a  human  in  the  loop  experiment  focused  on  knowledge  elicitation  and  direction  of 
attention  for  a  human  observer  [15],  and 

(9)  Initial  demonstrations  to  illustrate  the  use  of  Twitter  for  data  collection  and  reporting  ([16],  [26],  [31]). 

Technology  transfer  occurred  via  hosted  workshops,  special  sessions  in  International  Information  Fusion 
conferences,  meetings  with  government  and  industry  personnel  and  technical  reports  for  the  Joint  IED  Defeat 
Organization  (JIEDDO)  and  the  National  Geospatial  Intelligence  Agency.  A  summary  of  these  areas  is  presented 
below. 

2.1  Human-Centered  Information  Fusion 

The  concept  of  hard  and  soft  information  fusion  may  be  viewed  as  part  of  the  larger  concept  of  human -centered 
information  fusion  described  by  Hall  and  Jordan  [1],  The  conventional  focus  of  the  data  fusion  community  has 
been  the  use  physical  sensor  sources  such  as  visual  and  infrared  imagery,  radar,  satellite,  and  acoustic  sensor  data  to 
observe  physical  entities  like  troops,  vehicles,  weapon  systems,  or  other  objects.  Historically,  this  type  of  sensor 
activity  has  been  very  helpful  for  performing  tracking,  situation  assessment,  and  threat  assessment  for  military 
operations.  Two  recent  factors  have  caused  a  major  reassessment  of  this  paradigm.  First,  military  emphasis  has 


largely  shifted  from  conventional  warfare  to  the  challenges  of  counterinsurgency  and  counterterrorism.  Second,  the 
emerging  concept  of  human-centered  information  fusion  [1]  is  causing  the  exploration  of  new  ways  in  which 
humans  and  computer  systems  can  work  together  to  address  challenges  in  a  manner  that  optimally  utilizes  the 
capabilities  of  physical  sensors,  computer  hardware  and  software,  supporting  cyber-infrastructure,  and  human  beings 
([2],  [4]). 

A  particular  focus  of  evolving  fusion  concepts  involves  counter  insurgency  (COIN)  applications  such  as  the 
detection  and  defeat  of  Improvised  Explosive  Devices  (IEDs).  While  such  tasks  as  defeating  IEDs  still  rely  on 
observation  of  physical  entities  (such  as  explosives,  vehicles,  and  communication  devices),  it  also  requires  an  in- 
depth  understanding  of  the  social  networks,  intent,  belief  systems,  connectivity,  policies,  and  procedures  that  drive 
the  process.  Physical  sensors  can  be  useful  in  detecting  some  of  the  physical  manifestations  of  these  abstract 
concepts.  However,  they  are  largely  unable  to  effectively  classify  their  most  important  aspects  without  human 
intervention.  In  the  past,  the  human  role  in  the  fusion  process  has  largely  been  limited  to  analyzing  the  completed 
output  of  the  fusion  system.  Unfortunately,  when  the  human  ability  to  observe  and  ascertain  intent,  beliefs,  and 
cultural  influences  is  critical  to  the  sense-making  process,  any  fusion  system  that  withholds  the  role  of  the  human 
until  the  end  of  the  process  is  doomed  to  fail. 

The  concept  of  data  fusion  is  evolving  to  integrate  humans  in  a  variety  of  roles  [1],  First,  there  is  the  role  of  humans 
as  observers  (i.e.,  “soft  sensors”  or  participatory  sensing  ([4],  [5],  [6])).  Although  humans  are  not  able  to  compete 
with  conventional  hard  sensors  at  many  tasks,  humans  have  an  unparalleled  ability  to  instinctively  and  intuitively 
make  sense  of  complex  situations  and  interactions  that  would  leave  the  most  advanced  state-of-the-art  computerized 
sense-making  systems  at  a  loss.  With  the  assistance  of  ubiquitous  smart  phones  and  robust  data  networks,  human 
observers  have  the  capability  to  transmit  annotated,  geospatially  and  temporally  stamped  high -resolution  imagery 
and  video  nearly  instantly.  Additionally,  social  networking  tools  such  as  Twitter  facilitate  crowd-sourced  sensing 
that  can  be  either  tasked  or  opportunistic  [7].  When  tasked  sensing  is  performed,  the  collection  of  desired  data  is 
requested  of  the  human  observer.  Opportunistic  sensing  uses  data  that  was  gathered  and  annotated  for  other 
purposes  and  published  to  the  open  source  community.  The  concept  of  human  centric  fusion  is  shown  in  Figure  1. 


Conventional 

sensors 


Figure  1 :  Concept  of  Human-Centric  Information  Fusion 


Hall  and  Jordan  [1]  provide  a  detailed  description  of  the  concept  of  human  centric  information  fusion  and  discuss 
issues  related  to  all  of  the  human  roles  identified  above. 


2.2  Hard  and  Soft  Fusion  Frameworks 


Initial  research  in  the  area  of  hard  and  soft  fusion  was  in  part  the  result  of  this  topic  being  addressed  at  the  2008 
Critical  Issues  in  Information  Fusion  Workshop1.  At  that  workshop  general  processing  strategies  for  combining  and 
fusing  observational  data  from  hard  and  soft  data  inputs  were  discussed.  These  discussions  ranged  from  low-level 
(LI)  fusion  processing  operations  to  high-level  (L2,  L3)  fusion  processing.  Some  crucial  issues  in  deciding  on  a 
process  framework  of  course  are  how  to  both  represent  and  quantify  the  confidence  in  the  soft  data  but  also  how  to 
incorporate  the  distinctive  inputs  that  a  human  observer  can  provide,  e.g.  assessments  of  relationships,  judgments  of 
various  type,  and  estimates  of  emotional  states  etc.  It  is  important  in  exploring  notions  of  processing  frameworks 
not  to  consider  human  observational  capabilities  in  the  context  of  a  “sensor  surrogate”  but  rather  to  exploit  these 
distinct  human-only  skills.  The  other  major  distinction  in  soft  data/human  observation  is  that  the  observational 
“content”  is  represented  in  natural  language,  imputing  all  of  the  complexities  in  processing  and  understanding 
natural  language  forms. 


Upon  consideration  of  these  and  many  other  factors  that  distinguish  hard  data  from  electronic/physics -based  sensors 
and  soft  data  from  human  observers  and  yet  other  data  in  linguistic  form  (e.g.  much  of  contextual  information,  Web- 
service-based  inputs,  etc.),  an  approach  was  formulated  as  shown  in  Figure  2  (and  used  in  a  successful  joint 
University  at  Buffalo-Penn  State  Army  Research  Office  Multidisciplinary  University  Research  Initiative  (MURI) 
proposal  on  Hard-Soft  Data  Fusion). 


Figure  2:  Hard  and  Soft  Data  Fusion  Framework 


The  foundational  arguments  for  this  approach  are  as  follows: 


1  2008  Critical  Issues  in  Information  Fusion  Workshop  hosted  by  the  Center  for  Multisource  Information  Fusion, 
University  at  Buffalo/CUBRC,  October  2008 


The  general  idea  is  to  conduct  overall  hard  +  soft  fusion  at  the  state/estimate  level  versus  measurement 
level 


—  Allows  straightforward  exploitation  of  existing,  data-class-specific  team  algorithms  and  methods 

—  Allows  data-specific  disparate  algorithms  to  interoperate  in  a  hybrid,  multi-technique  approach 

•  Frame  as  services;  analogs  to  existing  legacy  capabilities 

•  Smooth  porting  to  network  environment 

—  Avoids  data  association  with  disparate  observational  data 

—  Fuses  at  the  common-semantics  level  (~  Decision-level  Fusion) 

•  Lower  computational  load 

•  Lower  communication  load  (looking  ahead  to  network-based  studies) 


The  framework  shown  in  figure  2  is  conceptual.  Additional  levels  of  detail  must  be  developed  to  provide  a  basis 
for  algorithm  selection,  implementation  and  demonstration.  An  example  of  a  further  refinement  of  the  framework 
is  shown  in  figure  3.  In  that  figure,  three  parallel  processing  flows  are  shown:  (1)  the  top  part  of  the  figure  shows 
the  processing  of  human  reports  (with  functions  such  as  message  formulation,  word-sense  disambiguation, 
automated  filtering,  soft  source  characterization,  focus  of  attention/knowledge  elicitation  aids,  etc.;  (2)  the  central 
part  of  the  figure  illustrates  the  typical  processes  required  for  ingesting  hard  sensor  data  including  common 
functions  such  as  signal  conditioning,  feature  extraction,  common  referencing,  inter-source  association,  etc.;  (3) 
Finally,  the  bottom  part  of  the  figure  shows  the  type  of  processes  required  to  address  web -based  information,  using 
search  engines  in  effect  as  “web-source  observers”.  A  further  refinement  of  this  processing  flow  is  introduced  in 
section  2.4  of  this  report. 
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Figure  3:  Example  of  Processing  Flow  for  Hard/Soft  Fusion 


Another  approach  to  developing  a  functional  framework  (and  identifying  relevant  models  and  algorithms)  is  to 
follow  the  Joint  Directors  of  Laboratories  (JDL)  data  fusion  process  model  ([32],  [18])  and  explore  an  analog 
between  traditional  fusion  processing  (at  the  JDL  level  0  and  level  1  sub-processes)  for  physical  sensors  observing 
physical  targets,  and  creating  similar  sub-processes  and  functions  for  soft  sensors  observing  the  human  landscape. 
Such  an  assessment  is  shown  in  figure  4.  In  that  figure,  the  left  hand  side  of  the  figure  shows  the  level  0  and  level  1 
processes  for  translating  traditional  hard  sensor  data  (e.g.,  signal  and  image  data)  about  physical  targets  to  state 
vectors  that  represent  or  characterize  the  targets  (e.g.,  position,  velocity,  identity,  attributes,  etc.).  The  types  of 
functions  include  data  abstraction  and  representation,  source  characterization  and  meta-data  generation,  co¬ 
referencing,  data  association  and  correlation  and  finally  state  estimation.  Types  of  algorithms  and  models  are 
shown  on  the  far  left  hand  side  of  the  figure.  By  contrast,  the  right  hand  side  of  the  figure  shows  the  parallel 
functions  and  algorithms  for  soft  sensors  observing  the  human  landscape.  In  this  case  much  of  the  data  involves 
text-based  information  which  still  requires  functions  such  as  data  abstraction,  source  characterization,  co¬ 
referencing,  data  association  and  correlation  and  estimation.  However,  the  specific  types  of  algorithms  shown  on 
the  far  right  hand  side  of  the  figure  are  different  than  those  used  for  processing  hard  sensor  data.  This  is  to  be 
expected.  However,  it  should  also  be  clear  that,  as  in  processing  of  hard  sensor  data  for  the  physical  landscape, 
there  are  no  “magic  algorithms”  that  address  all  aspects  of  modeling  and  prediction  for  the  human  landscape. 


•  Models  of  state  dynamics  (behavior) 

•  Models  of  observation  to  state 

•  Models  of  environmental  effects 

•  Models  of  error  growth 


Genera  lly  sea  lar  to  sea  la  i 
scalar  to  vector  or  vector 
vector  association 


irto  ^ 


Estimation 

Data  Association/Correlation 

(Hypothesis  Generation,  Hypothesis  Evaluation,  Hypothesis  Selection) 


'  Reasoning  at  Level-2  and  Level-3 
■  Models  of  human  behav  ior  (forward  propagation;  ofcomple 
“state"  |e.g.  large  social  net) 

•  Need  link  between  observables  &  “state* 

•  issues  of  hierarchical  levels  of  "granularity*  in  state 

•  Graph  dynamic  models 

-  •  Agent-based  models,  etc. 

{Partial  graph  matching 

•  vector  to  graph  or  graph  to  graph  assoc 
•  Asynchronous  temporal  correlation 
•  Semantic  distance,  generalized  measures? 


•  Conflation 

•  Platform/sensor  modeling 


Characterize  Sensor/source 

•  Environment 
•Training  set  (ID) 

•  Target  characteristics 

•  Level  of  maintenance 

•  Calibration 
•Etc 

•Automated  semantic 
labeling 

•Pattern  recognition 
•Feature  extraction 


GIS  referencing  (relative  to  references 
location/observer) 

Asynchronous  temporal  correlation 
Namede-confiiction  (alias  processing) 


Source  Characterization 
&  Meta-data  generation 


D  at  a  Ab  st  racti  o  n/re  p  re  se  ntat  i  o  n 


Environmental. 

Conditions 

Mission 


Source  Characterization 
&  Meta-data  generation 


Data  Abstraction/representation 


A/ 


Sensor  Networks 


Real  World/Situation  (Physical,  Human  &  Cyber  Landscapes) 


Characterize  observer  (s) 

•  Level  of  training 

■  Type  of  target/situation 

•  Stress/mood 

•  NTC  lessons  learned 

•  Deception  analysis 

•  Parse  natural  language 

•  Semantic  transformations 
•Thesaurus  access 

•  Arias  processing 
•Ontology  access 

•  Fuzzy  logic  (de-fuzzification) 

•Tag  clouds 

•Computer  aided  search/summarization  & 
semantic  pattern  analysis 


Etc 


Knowledge  elicitation 
•interrogation 

•  interviews 

•  Ad  hoc  "crowd-sourcing* 

•  Structured  MCls, 

•  Storification 

•  Need  to  collect  data  &  meta-data 
simultaneously 


Figure  4:  Information  Fusion  Hierarchy  (JDL  Levels  0  &  1)  for  Hard  and  Soft  Data 

2.3  Literature  Review  of  Modeling  Techniques  for  Representing  Human  Behavior 

In  conjunction  with  related  tasking  on  a  National  Geospatial  Intelligence  Agency  (NGA)-sponsored  research  task, 
efforts  were  combined  to  produce  a  literature  survey  report  on  “Understanding  and  modeling  the  New  Domains”, 
meaning  a  survey  of  research  on  modeling  adversarial  human  behaviors,  typically  in  asymmetric,  irregular-type 
warfare  environments.  It  is  these  problem  domains  for  which  hard  and  soft  data  are  most  relevant. 

The  list  of  references  analyzed  in  this  work  is  shown  below.  A  diagrammatic  summary  of  the  analysis  is  shown  in 
Figures  5  and  6. 


•Modeling  of  Social  Dynamics 
•Human  Activity  Modeling 
•Human  Performance  Modeling 
•Artificial  Societies 
•Virtual  Humans 

•Pattern  Learning  from  Social  Network 
Analysis 

•Modeling  Crowd  Behavior,  Dynamics 
•Group  Dynamics 

•Computer-generated  Forces/Modeling 
Terrorist  Networks 
•Modeling  Terrorist  Decision-patterns 
•Terrorism  and  Game  Theory 
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Figure  5:  Surveyed  Categories  of  Modeling  and  Overall  Modeling  Capability  Assessment 

The  names  of  the  many  different  modeling  techniques  found  in  the  literature  are  shown  on  the  left  side  of  Figure  5. 
Each  of  these  research  areas  in  turn  apply  a  variety  of  modeling  paradigms  in  their  work  (see  Figure  6  below)  but  in 
spite  of  these  extensive  efforts,  the  assessments  conducted  for  example  by  the  National  Research  Council  (Ref 
Zacharias  2008  below)  indicate  that  the  state  of  the  art  is  still  at  a  very  limited  level  of  capability.  This  is  why,  in 
Figure  5  above,  we  judge  the  overall  capability,  in  NASA  TRL  terms,  as  Level  2,  requiring  further  exploration. 
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FIGURE  II-I  Perceived  similarities  among  types  of  models. 


Figure  6:  Variety  of  Mathematical  Models  Applied  to  Fluman  Behavior  Modeling  (Ref  Zacharias  2008  ) 


The  following  are  key  references  for  the  Human  Behavior  Modeling  Survey: 


Wray,  R.E.,  “Variability  in  Human  Behavior  Modeling  for  Military  Simulations”,  Presented  at:  Behavior 
Representation  in  Modeling  &  Simulation  Conference  (BRIMS).  May,  2003 

Wise,  B.P.,  et  al,  “Task  Order  (TO)  69  ATM  Human  Behavior  Modeling  Approach  Study”,  Technical 
Research  in  Advanced  Air  Transportation  Concepts  &  Technologies  (AATT),  SAIC  Report  Nov  2001  (for 
NASA) 

WMD  Terrorism  Project,  Center  for  Nonproliferation  Studies,  Monterey  Institute  of  International  Studies, 

CA.,  “Literature  Review  of  Existing  Terrorist  Behavior  Modeling”,  Final  Report  to  the  Defense  Threat 
Reduction  Agency,  14th  August,  2002 

Simulation  Interoperability  Standards  Organization:  Conference  Proceedings:  Computer-Generated  Forces  and 
Behavior  Representation  Conferences;  there  have  been  1 1  of  these  through  and  including  2009;  see  the  index 
at  www. sisostds.org/conference/index. cfm?conf=  1  lcgf 

Zacharias,  G.L.,  MacMillan,  J.,  and  Van  Hemel,  S.B.,  Editors,  Committee  on  Organizational  Modeling  from 
Individuals  to  Societies,  “Behavioral  Modeling  and  Simulation:  From  Individuals  to  Societies”,  National 
Research  Council,  2008 

Pew,  R.W.  and  Mavor,  A.S.,  Editors,  Panel  on  Modeling  Human  Behavior  and  Command  Decision  Making: 
Representations  for  Military  Simulations,  Committee  on  Human  Factors,  Commission  on  Behavioral  and 
Social  Sciences  and  Education,  “Representing  Human  Behavior  in  Military  Simulations”,  Interim  Report, 
National  Research  Council  NATIONAL  ACADEMY  PRESS,  Washington,  D.C.,  1997 

Pew,  R.W.  and  Mavor,  A.S.,  Editors,  Panel  on  Modeling  Human  Behavior  and  Command  Decision  Making: 
Representations  for  Military  Simulations,  Committee  on  Human  Factors,  Commission  on  Behavioral  and 
Social  Sciences  and  Education,  “Modeling  Human  and  Organizational  Behavior:  Application  to  Military 
Simulations”,  NATIONAL  ACADEMY  PRESS,  Washington,  D.C.  1998 

Silverman,  B.G.,  et  al,  “Human  Behavior  Models  for  Agents  in  Simulators  and  Games”,  Presence: 
Teleoperators  and  Virtual  Environments,  Volume  15,  Issue  2,  April  2006 

Weaver,  R.,  et  al,  “Modeling  and  Simulating  Terrorist  Decision-making:  A  “Performance  Moderator  Function” 
Approach  to  Generating  Virtual  Opponents”,  Proceedings  of  the  10th  Conference  On  Computer  Generated 
Forces  and  Behavioral  Representation,  May  2001. 

Goldstone,  R.L.  and  Janssen,  M.A.,  “Computational  Models  of  Collective  Behavior”,  TRENDS  in  Cognitive 
Sciences  Vol.9  No. 9  September  2005 

Warwick,  W.,  et  al,  “Developing  Computational  Models  of  Recognition-Primed  Decision  Making”,  10th  Conf 
on  Computer-generated  Forces,  May  2001 

Mannes,  A.,  et  al,  "Stochastic  Opponent  Modeling  Agents:  A  Case  Study  with  Hezbollah,"  First  International 
Workshop  on  Social  Computing,  Behavioral  Modeling,  and  Prediction,  April  2008 

Goerger,  S.R.,  “Validating  Computational  Human  Behavior  Models:  Consistency  and  Accuracy  Issues”,  PhD 
Thesis,  Naval  Postgraduate  School,  2004 

van  Doesburg,  W.A.,  “Quality  Assessment  of  Human  Behavior  Models”,  in  Proc.  Artificial  Intelligence  and 
Soft  Computing,  Spain,  2007 


2.4  Theoretical  Framework  for  Fusion  of  Hard  and  Soft  Data 


The  definition  and  selection  of  algorithms  for  hard  and  soft  fusion  continues  to  evolve.  A  key  aspect  of  the  on¬ 
going  ARO  MURI  project  on  hard  and  soft  fusion  is  focused  on  developing  and  refining  such  a  framework.  At  this 
time,  algorithms  and  techniques  are  being  identified  and  prototyped  ([6],  [8],  [13],  [27],  [28]).  The  initial  strategy 
for  fusion  has  been  to  separate  the  hard  and  soft  information  processing  flows  and  to  perform  multi -source  fusion  for 
each  information  stream,  with  subsequent  fusion  of  hard  and  soft  processing  results  at  the  report  level.  That  is,  hard 
sensor  data  are  fused  to  result  in  a  series  of  reports  or  state  vectors  concerning  observed  activities,  events,  or  entities. 
Hence,  the  hard  fusion  process  transforms  observed  energy  from  physical  sensors  into  a  generalized  state  vector 
which  may  include  semantic  information  (e.g.,  representing  an  assertion  of  entity  identification  based  on  pattern 
recognition  techniques),  scalar  and  vector  data  describing  the  characteristics,  attributes  and  location/kinematics  of 
observed  entities.  Similarly,  soft  data  observations  (which  typically  are  asserted  at  the  semantic  level)  are 
transformed  into  a  general  state  vector  characterizing  an  observed  entity,  activity  or  event.  For  soft  data,  semantic 
assertions  of  location  (e.g.,  “the  car  is  near  the  mosque”)  may  require  translation  to  a  scalar  measure  via  de¬ 
fuzzification  algorithms  [33],  Subsequently  these  general  state  vectors  or  reports  are  fused  using  methods  such  as 
Bayesian  Belief  Nets,  Generalized  Logical  Templates,  or  Graph  Matching  methods.  The  conceptual  framework  is 
illustrated  in  Figure  7. 

We  note  that  the  standard  requirements  for  source  calibration/characterization,  uncertainty  representation, 
association  and  correlation  and  state  estimation  must  be  addressed  for  the  hard  fusion  processing  stream,  the  soft 
fusion  processing  stream  and  for  the  hard/soft  fusion  processing.  It  is  conceivable  that  fusion  can  be  performed  at 
the  feature  or  data  level  (viz.  seeking  to  fuse  the  hard  and  soft  data  “closer”  to  the  observations),  although  there  are 
a  number  of  potential  modeling  problems  that  would  need  to  be  addressed. 
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Figure  7 :  Evolving  Architecture  for  Hard/Soft  Data  Fusion 


2.5  Literature  Review  of  Human  Observational  Capabilities 

In  this  task,  we  leveraged  NGA-sponsored  tasking  to  produce  a  report  for  Penn  State  on  Human  Observational 
capabilities.  The  main  result  of  this  review  shows  that  no  general  models  seem  to  exist;  many  pointed  studies 
regarding  human  observational  capabilities  have  been  done  (to  include  Army-sponsored  research)  but  all  have  been 
very  directed,  involving  very  specific  observational  conditions  and  observational  goals.  The  works  can  be 
summarized  in  the  following  tables,  which  include  the  cited  references. 

Additional  analysis  and  literature  review  is  described  by  Hall  and  Iordan  [1]  in  chapter  3.  They  use  a  conceptual 
framework  as  illustrated  in  Figure  8  as  a  basis  for  reviewing  the  literature  on  human  observational  capabilities  and 
limitations. 


Figure  8:  Conceptual  Framework  for  Understanding 
Human  Observations  [1] 
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Sample  Research  Findings 

Reference 

Apparent  size 

Apparent  size  with  distance  depends  on 
viewing  mode  and  the  angle  subtended  by 
the  object;  if  the  subtended  angle  is  fixed, 
the  apparent  size  is  fixed. 

Determinants  of  Apparent  Visual  Size  with  Distance 
Variant  Author(s):  Alfred  H.  Holway  and  Edwin  G.  Boring 
Source:  The  American  Journal  of  Psychology,  Vol.  54,  No. 

1  (Jan.,  1941),  pp.  21-37 

Change  blindness 

At  any  instant,  little  visual  information  is 
retained,  and,  over  time  and  across 
fixations  the  representation  does  not  get 
more  complete  and  complex.  Instead,  as 
attention  shifts,  so  does  the  content  of  the 
mental  representation  (volatility  of 
mental  representations). 

Volatile  visual  representations:  Failing  to  detect  changes 
in  recently  processed  information 

M.  W.  Becker  and  H.  Pashler,  University  of  California, 

San  Diego,  La  Jolla,  California,  Psycho nomic  Bulletin  & 
Review,  2002, 9  (4),  744-750 

Disinformation 

Detection 

Tools  seem  to  exist  for  helping  to  detect 
disinformation  (in  documents) — not 
perfectly  but  they  reduce  human 
workload.  One  example:  the  Linguistic 
Pattern  Analyzer  (LPA)  compiles  and 
scores  information  directly  related  to 
indicators  of  multiple  computational 
social  sciences  models  of  various 
behaviors  to  include  deception. 

Models  of  Trust  and  Disinformation  in  the  Open  Press, 
from  Model-Driven  Linguistic  Pattern  Analysis,  G.  A. 
Mack,  S.  G.  Eick,  M.  A.  Clark,  Aerospace  Conference, 
2007  IEEE 

March  2007 

Distance 

Perception 

Peop  le  are  good  at  estimating  distance  if 
the  observational  setting  is  static  and 
level.  When  moving,  the  estimates  can 
become  better  due  to  the  continuing 
changes  in  perspective.  But  distance 
estimates  are  also  affected  by  emotional 
and  other  factors. 

Range  estimation  in  various  settings 
show  a  power-law  relationship  between 
estimated  and  actual  range  with  the 
exponent  varying  between  about  0.8  and 
1.25 

Distance  Perception 

D.  R.  Proffitt,  Current  Directions  in  Psychological 

Science,  Volume  15,  Number  3, 2006 

Range  estimates  of  distant  visual  stimuli,  E.  Galanter  and 
P.  Galanter,  Perception  &  Psychophysics,  1973,  Vol.  14, 
No.  2 

Deception 

Detection 

This  and  other  papers  substantiate  that 
humans  have  p  oor  ability  to  detect  lies 
and  deception;  scores  are  typically  in  the 
55-60%  range. 

Human  ability  to  detect  verbal  and  non-verbal  deception, 

ivythesis.typepad.com/term_paper_topics/2008/02/the- 

human-abili.html 

Training  effects 
on  distance 
perception 

Effects  of  training  do  improve  accuracy 
but  only  slightly 

The  Effect  of  Prior  Training  with  a  Scale  of 

Distance  on  Absolute  and  Relative  Judgments  of 

Distance  Over  Ground, 

E.  J.  Gibson,  R.  Bergman,  and  J.  Purdy,  J  Exp  Psychol. 
1955  Aug 

Interobject 

Distances 

Interobject  distances  are  generally 
overestimated 

Factors  Affecting  the  Perception  of  Interobject  Distances 
in  Virtual  Environments,  D.  Waller,  Presence,  Vol.  8, 

No. 6,  December  1999 

Informant  and 

rumor 

information 

Various  models  exist,  some  quantitative  ; 
in  general  there  is  a  condensation  to  a 
few  key  factors  that  are  reported 
reasonably  consistently — details  are  lost 
however 

Information  Transmission  Through  Human  Informants: 
Simulation  1 

G.  C.  Lawson  and  C.  T.  Butts,  Proceeding  of  NAACSOS 
2004,  K.  Carley  (Ed.). 

Velocity 

differences 

Within  an  optimal  velocity  range  humans 
can  p  erform 

accurately  in  a  speed  discrimination  task 
with  a  velocity  difference  as  little  as  7%. 
But  for  motion  faster  or  slower  than  this 
the  brain  cannot  discriminate  the  faster 
of  two  objects  so  accurately,  even  if  the 
difference  is  quite  large. 

Illusions  disrupting  the  accurate  perception  of  velocity 
and  position,  L.  Dartnell 
www.ucl.ac.uk/~u  cbplrd/Lewis_Dartnell- 
MRes_project.pdf 

2.6  Algorithms  for  Fusion  of  Hard  and  Soft  Data 

Many  of  the  fusion  algorithms  applicable  to  traditional  hard  sensor  fusion  [18],  [19])  remain  applicable  to  hard  and 
soft  information  fusion.  As  illustrated  in  Figure  7,  basic  functions  such  as  source  characterization,  association  and 
correlation,  hypothesis  generation  and  selection,  pattern  recognition,  etc.  are  requisite  functions  for  hard  and  soft 
fusion.  Initial  work  has  been  conducted  by  Sambhoos,  Llinas  and  Little  [8]  and  Sambhoos  et  al  [13].  This  work 
has  focused  on  graph  matching  techniques.  Other  researchers  at  Penn  State  are  developing  methods  for  fusing  hard 
sensor  data  involving  the  combination  of  information  from  3-D  active  sensors  (e.g.,  LIDAR)  with  2-D  image  sensor 
data  and  with  acoustic  data  [27].  Additional  work  is  exploring  the  use  of  intelligent  agent  formulations  [28], 

Researchers  at  the  University  at  Buffalo  are  focusing  on  soft  sensor  processing  algorithms  at  the  semantic  and  report 
level.  A  key  component  of  the  ARO  MURI  program  on  hard  and  soft  fusion  is  to  implement  and  explore  such 
techniques.  In  this  project,  we  established  the  basic  framework  and  information  architecture  for  selection  and 
implementation  of  soft/hard  fusion  methods.  A  challenge  in  hard  and  soft  fusion  involves  how  to  address  issues  of 


the  diversity  of  uncertainty  representation  (e.g.,  the  use  of  probabilistic  formulations  for  hard  sensor  processing  and 
the  use  of  fuzzy  membership  types  of  relationships  for  soft  fusion  processing). 

2.7  Extreme  Events  Laboratory 

The  Extreme  Events  Lab  (EEL)  at  the  Pennsylvania  State  University  was  designed  to  facilitate  research  in  hard  and 
soft  data  fusion,  visualization,  and  sonification  [11].  The  EEL  features  4  large  wall-mounted  High  Definition  Liquid 
Crystal  Display  displays,  2D  and  true  stereoscopic  3D  projectors,  a  stereoscopic  3D  Head-Mounted  Display  (HMD), 
haptic  input  devices,  motion  and  acoustic  sensors,  various  video  and  still  cameras,  several  MacPro  workstations,  a 
conference  area  with  a  54"  HD  LCD  TV,  and  8  professional-quality  loudspeakers  for  experimentation  with 
immersive  3D  sonification. 

The  hardware  can  be  easily  switched  between  configurations  optimized  for  maximum  display  capability  (e.g.  video - 
intensive  multi-modal  immersive  visualizations)  and  maximum  processing  power  (e.g.  complex  plume  or  geospatial 
modeling).  The  facility  can  be  used  to  test  Human  Computer  Interaction  methods  that  optimize  situational 
awareness  through  the  use  of  non-conventional  controller  interfaces  and  immersive  stereoscopic  viewing  of 
visualizations  and  geospatial  situations.  A  view  of  the  Extreme  Events  Lab  data  analysis  centre  is  shown  in  Figure 
9. 

The  stereoscopic  HMD  features  semi-transmissive  optics,  which  allows  the  computer -generated  3D  images  to  be 
superimposed  over  the  user’s  actual  field  of  vision  in  order  to  function  as  a  Heads-Up  Display  (HUD). 


Figure  9:  Interior  of  the  Extreme  Events  Lab 


2.8  Methods  for  Test  and  Evaluation  of  Hard/Soft  Fusion 

Waltz  and  Llinas  initially  described  the  challenges  of  test  and  evaluation  of  data  fusion  systems  in  their  landmark 
text  [22].  They  describe  a  hierarchy  of  evaluation  from  measures  of  performance  (MOP)  (e.g.,  the  receiver 
operating  characteristics  (ROC)  curves  for  individual  sensors)  to  measures  of  performance  (MOP)  (e.g.,  for  target 
trackers),  to  measures  of  effectiveness  (MOE)  and  ultimately  measures  of  force  effectiveness  (MOFE).  Evaluation 
of  the  hierarchy  of  fusion  system  performance  from  the  individual  physical  sensor  to  support  for  decision  making 
and  action  requires  extensive  data  sets  with  known  “ground  truth”.  Throughout  the  fusion  community  enormous 
efforts  have  been  made  to  collect  data  sets  in  laboratory  environments,  field  test  conditions,  development  of 


simulators  and  data  obtained  during  real  operations  [19].  Despite  these  efforts,  no  general  purpose  test  bed  exists  to 
evaluate  fusion  algorithms  for  hard  sensor  processing,  let  alone  for  hard  and  soft  fusion. 

The  need  for  calibrated  data  sets  is  exacerbated  for  human-centric  information  fusion  and  for  soft  and  hard  data 
fusion.  For  human-centric  fusion  we  must  consider  several  possible  roles  of  humans  (i.e.,  as  observers,  pattern 
recognizers/semantic  level  reasoners,  and  as  collaborative  decision  makers).  This  involves  all  of  the  issues 
associated  with  hard  sensor  processing  as  well  as  involving  how  to  evaluate  the  role  and  performance  of  humans  in 
the  various  roles.  Each  role  has  its  own  challenges  and  issues.  For  example,  the  human  as  observer  requires  the 
development  of  an  equivalent  “ROC”  curve  for  humans,  while  the  role  of  pattern  recognizer/semantic  level  reasoned 
involves  issues  of  perception,  attention,  cognition,  etc.  Finally,  collaborative  decision  making  involves  issues  of 
team  cognition  and  effective  collaboration.  Researchers  such  as  Ensley  [23]  and  McNeese  et  al  ([24],  [25])  have 
discussed  such  issues  in  detail  and  McNeese  has  developed  a  “living  laboratory”  approach  illustrated  in  Figure  10. 
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Figure  10:  Living  Laboratory  Approach  for  Human  in  the 
Loop  Experiments 

An  issue  for  hard  and  soft  fusion  test  and  evaluation  is  the  need  to  obtain  simultaneous  and  integrated  hard  data  and 
soft  data  about  a  domain  of  interest.  A  general  discussion  of  this  problem  for  hard  and  soft  fusion  is  provided  by 
Hall  et  al  [14].  Figure  1 1  shows  the  concept  of  a  hierarchy  of  potential  test  data  ranging  from  real  operational  data 
collected  in  field  operations  (in  the  upper  left  hand  side  of  the  figure),  to  data  from  training  exercises  [29],  to 
synthetic  hard  and  soft  data  created  especially  for  test  purposes  [30],  and  finally  the  conduct  of  campus  experiments 
to  illustrate  various  issues  such  as  tasking  of  human  observers,  dynamic  knowledge  elicitation,  training  of  observers, 
etc.  Beyond  this  project,  we  are  developing  such  data  sets  and  planning  relevant  campus  experiments  for  COIN 
applications. 


Real  Operational  Data 

•  Real  data  (most 
realistic) 

•  Classified 

•  Not  repeatable 

•  Not  collected  for 
T&E  purposes 


HASTEN  Data 

•  Real  data  but  / 
unknown  ground 
truth 

•  Distribution 
restrictions 

•  Not  repeatable 

•  Collected  for 
training  purposes 


PSU/IST  COIN 
Synthetic  Data  Set 

•  Developed  for  T&E 

•  Realistic  (truth  ^ 
based) 

•  Modifiableto  meet 
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coordinated  hard& 
soft  data 


Campus  Experiments 
and  Demonstrations 

•Analogs  of  COIN 
operations 

•  Dynamic 

•  Emulate  environment 
with  human  in  loop 

•  Repeatable 


Figure  1 1 :  Hierarchy  of  Test  Data  for  Soft/Hard  Fusion 


2.9  Human  in  the  Loop  Experiments 


During  this  project  some  limited  initial  human  in  the  loop  experiments/demonstrations  were  performed. 

Issues  in  the  utilization  of  humans  as  soft  sensors  involves  knowledge  elicitation  and  focus  of  attention.  Numerous 
examples  are  available  of  the  failure  of  human  observers  to  notice  critical  information,  or  conversely  to  report 
mistaken  observations.  An  example  of  the  former  is  the  failure  of  students  at  Virginia  Tech  University  to  observe 
the  precursors  to  the  shooting  spree  in  which  a  23  year  old  student,  Seung-Hui  Cho  killed  27  students,  five  teachers 
and  himself.  Observable  precursors  included  Cho  practicing  locking  a  building  with  chains  (to  prevent  police  from 
entering  and  stopping  his  shootings),  two  days  prior  to  the  actual  event.  By  contrast,  Kaplan  and  Kaplan  cite  the 
following  example  (p  82)  ;  In  1978  the  Rotterdam  zoo  reported  the  escape  of  one  of  its  red  pandas;  hundreds  of 
helpful  people  called  in,  having  spotted  it  in  places  all  over  the  Netherlands  -  when  in  fact  it  had  been  run  over  by  a 
train  just  a  few  yards  from  the  zoo  fence.”  Other  examples  of  challenges  in  human  observing  and  focus  of  attention 
include  experiments  involving  the  use  of  distractions.  In  an  experiment  conducted  by  D.  Simmons  and  C.  F. 
Chabris  at  the  University  of  Illinois,  for  example,  subjects  were  shown  a  video  clip  of  two  teams  of  people  (in  white 
and  black  uniforms)  passing  a  basketball  back  and  forth.  The  subjects  were  asked  to  count  the  number  of  passes  by 
one  team  during  a  60  second  period.  During  the  video,  a  person  in  a  gorilla  suit  walks  between  the  players,  stops 
and  waves  and  continues  on.  A  surprisingly  high  number  of  subjects  fail  to  notice  the  gorilla 
(http://viscog.beckman.illinois.edu/flashmovie/15.php).  Thus,  part  of  the  effort  associated  with  evaluating  the  use 
of  humans  as  observers  is  to  understand  issues  such  as  focus  of  attention,  knowledge  elicitation,  observer  bias  and 
training  and  other  effects. 

During  this  project,  a  master’s  degree  student,  Alice  Shapiro,  designed  and  conducted  and  experiment  focused  on 
how  to  guide  an  observer’s  awareness  for  detecting  an  unusual  person  in  a  crowd  [15].  Shapiro  developed  a  spatial 
attention  allocation  guide  to  assist  in  directing  an  observer’s  attention.  The  tool  involved  using  a  spatial  guide  (a 
circular  area  of  focus)  for  observing  video  clips.  Shapiro’s  research;  (1)  designed  a  spatial  visual  attention  guide, 
(2)  created  a  test  and  evaluation  video  footage  for  a  crowd  near  a  building  on  the  Penn  State  University  Park 
campus,  (3)  conducted  a  statistically  significant  experiment  involving  requesting  observers  to  find  a  specific  person 
(carrying  a  bag)  in  a  crowd  during  a  several  minute  sequence,  (4)  evaluated  the  effectiveness  of  the  attention  aid 
(accuracy  of  observation  with  and  without  the  aid),  and  (5)  evaluated  factors  such  as  observer  knowledge  of  the  area, 
experience  with  video  games  and  personal  attributes.  An  example  of  the  observer  attention  guide  is  shown  in 
Figure  12. 


Figure  12:  Example  of  visual  attention  aid  for  observation 


NC2IF  CAMPUS  EXPERIMENT:  PHASE  I 
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Figure  13:  Penn  State  campus  human  in  the  loop  experiment  concept 


A  number  of  human  in  the  loop  experiments  are  planned  at  the  Penn  State  University  Park  Campus  later  this  year  to 
illustrate  issues  such  as  tasking  of  observers,  dynamic  knowledge  elicitation  and  other  effects.  Figure  13  illustrates 
the  planned  locations  of  physical  sensors  and  use  of  student  observers  to  simulate  counter  insurgency  (COIN) 
activities  and  events.  Data  collected  will  include  the  “ground  truth”  of  events,  activities,  entity  locations,  identities 
of  scenario  actors,  timelines  of  unfolding  events  and  data  from  physical  sensors  as  well  as  human  reports.  Some 
initial  activities  have  involved  the  development  of  special  applications  (APPs)  for  mobile  phones  and  the  use  of 
group  game  playing  motivation  to  stimulate  competitive  collection  of  information  (e.g.,  about  potential  campus 
hazards).  The  concept  of  making  a  game-like  environment  to  stimulate  human  interest  is  sometimes  termed 
“gamification”.  Initial  experiments  are  described  by  McGill  [26]. 

On  a  final  note,  it  is  possible  to  leverage  other  participatory  sensing  experiments  and  exercises  to  learn  about  issues 
and  models  for  human  observers.  One  example  involved  the  DARPA  Red  Balloon  contest  (see 

https://networkchallenge.darpa.mil/rules.aspx).  This  contest  sought  teams  to  develop  methods,  using  social 
network  concepts,  to  locate  and  identify  10  red  weather  balloons  placed  around  the  United  States  during  a  weekend 
experiment.  The  prize  for  the  first  team  to  successfully  verify  the  location  of  the  10  balloons  was  $  40,000.  Over 
3800  teams  registered  for  this  challenge  problem,  including  a  team  of  undergraduate  students  at  the  Penn  State 
University  College  of  Information  Sciences  and  Technology.  As  might  be  imagined,  there  were  a  large  number  of 
false  reports  during  the  one -day  exercise. 

The  Penn  State  team  finished  10lh  out  of  3800  teams.  The  Penn  State  team  used  two  parallel  strategies;  (1)  creation 
of  a  national  social  network  of  Penn  State  students  and  alumni  to  act  as  ad  hoc  observers,  with  web-based  and 
mobile  computing  mechanisms  for  reporting  observations,  and  (2)  use  of  a  strategy  involving  “watching  the 
watchers”  (viz.,  monitoring  reports  from  other  observers  via  Facebook,  Twitter,  etc),  and  seeking  to  verify  or  refute 
the  dynamic  reports  faster  than  others  could.  This  strategy  of  “watching  the  watchers”  actually  obtained  more 
information  than  the  use  of  social  network  based  tasking  and  reporting.  Such  strategies  might  have  implications  for 
COIN  applications.  A  description  of  the  experiment  and  lessons  learned  is  provided  by  [31]. 


3.0  Technology  Transfer 

A  summary  of  technology  transfer  mechanisms  and  events  is  provided  below. 

J.  Llinas  hosted  the  2008  Critical  Issues  in  Information  Fusion  Workshop  (via  the  Center  for  Multisource 
Information  Fusion,  University  at  Buffalo/CUBRC)  in  October,  2008  at  which  40  scientists  from  DoD  and 
industry  met  to  discuss  concepts  and  issues  in  hard  and  soft  information  fusion 

J.  Llinas  and  D.  Hall  assisted  in  organizing  and  hosting  special  sessions  at  the  1 1th  and  12th  International 
Conference  on  Information  Fusion  (in  July,  2008  in  Cologne,  Germany  and  in  July,  2009  in  Seattle, 
Washington  on  the  topics  of  hard  and  soft  data  fusion. 

D.  Hall  and  J.  Llinas  co-hosted  a  2-day  data  fusion  technology  workshop  on  December  6th  and  7lh  held  in 
State  College,  PA  for  government  representatives  from  industry  and  the  National  Geospatial  Intelligence 
Agency  (NGA)  to  discuss  technology  issues  in  information  fusion. 

D.  Hall  and  J.  Llinas  co-hosted  a  2-day  information  fusion  working  group  on  Future  Roles  of  Fusion 
Technology  to  Support  COIC  Operations,  for  the  Joint  Improvised  Explosive  Device  Defeat  Organization 
(JIEDDO)  in  August  2010  and  also  contributed  to  a  technical  report  entitled.  Future  Roles  of  Fusion 
Technology  to  Support  COIC  Operations  for  JIEDDO,  Sept  17,  2010 

Special  sessions  on  hard  and  soft  data  fusion  are  also  planned  at  the  International  Conference  on 
Information  Fusion  to  be  held  in  Chicago,  Illinois  in  July,  2011. 

4.0  Key  Personnel 

Key  personnel  on  this  project  include  Dr.  David  Hall  and  Dr.  James  Llinas.  Brief  bio-sketches  are  provided  below. 

Dr.  David  Hall  is  the  principal  investigator  on  this  project.  Dr.  Hall  is  a  Professor  in  the  College  of  Information 
Sciences  and  Technology  at  the  Pennsylvania  State  University.  He  also  directs  the  Penn  State  Center  for  Network 
Centric  Cognition  and  Information  Fusion  (NC2IF).  He  has  more  than  25  years  of  experience  in  research,  research 
management,  and  systems  development  in  both  industrial  and  academic  environments.  Dr.  Hall  has  performed 
research  in  a  wide  variety  of  areas  including  celestial  mechanics,  digital  signal  processing,  software  engineering, 
automated  reasoning,  and  multisensor  data  fusion.  During  the  past  15  years,  his  research  has  focused  on  multisensor 
data  fusion.  He  is  the  author  of  over  175  technical  papers,  reports,  book  chapters,  and  books.  Dr.  Hall  is  a  member 
of  the  Joint  Directors  of  Laboratories  (JDL)  Data  Fusion  Working  Group.  He  serves  on  the  Advisory  Board  of  the 
Data  Fusion  Center  based  at  the  State  University  of  New  York  at  Buffalo.  In  addition,  he  serves  on  the  National 
Aeronautics  and  Space  Administration  (NASA)  Aeronautics  and  Space  Transportation  Technology  Advisory 
Committee.  In  2001,  Dr.  Hall  was  awarded  the  Joe  Mignona  award  to  honor  his  contributions  as  a  national  leader  in 
the  Data  Fusion  Community.  The  Data  Fusion  Group  instituted  the  award  in  1994  to  honor  the  memory  of  Joseph 
Mignona.  Dr.  Hall  was  named  as  an  IEEE  Fellow  in  2003  for  his  research  in  data  fusion. 

Dr.  James  Llinas  is  the  co-Principal  Investigator  on  this  project.  Dr.  Llinas  brings  over  30  years  of  experience  in 
multi-source  information  processing  and  data  fusion  technology  to  his  research,  teaching,  and  business  development 
activities.  He  is  an  internationally-recognized  expert  in  sensor,  data,  and  information  fusion,  co-authored  the  first 
integrated  book  on  Multisensor  Data  Fusion,  and  has  lectured  internationally  for  about  20  years  on  this  topic.  He  is 
co-editor  of  the  recently-published  Handbook  of  Multisensor  Data  Fusion  (CRC  Press).  Dr.  Llinas  is  a  Technical 
Advisor  to  the  Defense  Department’s  Joint  Directors  of  Laboratories  Data  Fusion  Group,  the  only  US  DoD 
technology  oversight  group  for  Data  Fusion,  a  position  he  has  held  for  over  15  years.  Dr.  Llinas  was  awarded  the 
definitive  US  defense  community  award  from  the  Data  Fusion  community,  the  Joe  Mignona  Award,  in  1999.  In 
addition,  reflecting  his  international  interests  and  stature.  Dr.  Llinas  was  voted  as  the  first  President  of  the 
International  Society  for  Information  Fusion  in  1998.  Dr.  Llinas  created  the  concept  for  and  is  Executive  Director 
for  the  “Center  for  Multisource  Information  Fusion’’  located  at  the  State  University  of  New  York  at  Buffalo.  This 
first-of-its-kind.  University-based  research  has  received  sponsorship  from  a  broad  base  of  defense  and  industrial 
R&D  organizations,  and  is  conducting  basic  research  in  Distributed  Situational  Estimation,  Distributed  Learning, 
and  in  Correlation  Science,  among  other  programs. 


5.0  Leveraged  Projects 

This  project  leveraged  extensive  previous  and  on-going  research.  A  summary  is  provided  in  Table  3  below. 


Table  3:  Examples  of  On-Going  Research  Projects  related  to  Data  Fusion 


Project 

Sponsor 

Funding 

Principal 

Investigator 

Leveraged  Technology 

Multidisciplinary 
University  Research 
Initiative  (MURI)  on 
hard  and  soft  fusion 

ARO 

On¬ 

going 

J.  Llinas 

Detailed  investigation  of  fusion 
algorithms  and  development  of 
appropriate  test  and  evaluation 
data  sets 

Defense  University 
Research  Instrumentation 
Program  (DURIP) 

DoD 

$  200  K 

D.  Hall 

Infrastructure  for  Extreme  Events 
Laboratory 

Intelligent  distributed 
group  and  team  training 
systems 

AFOSR 

$  247  K 

J.  Yen 

Development  of  new  methods  to 
support  team  decision-making 
using  intelligent  agents  and 
collaboration 

Distributed  cyber¬ 
collaboration  and 
information  fusion 

ONR 

$  213  K 

D.  Hall 

Modeling  of  distributed,  dynamic 
sensor  networks  using  advanced 
approximations  for  probabilistic 
graph  methods 

Integrated  explanation 
and  visualization  of 
intelligent  synthetic 
forces 

Soar 

Technologies 

$  109  K 

F.  Ritter 

Cognitive  models  related  to 
explanation  and  visualization  of 
synthetic  forces 

Determine  USMC  anti¬ 
terrorism/force 
protection  for  resources 
and  facilities  planning 
and  design 

USMC 

$  433  K 

S.  Haynes 

Development  of  a  decision  model 
to  support  assessment  and 
tradeoffs  of  potential  threats  to 
military  resources  and  facilities  & 
planning  of  protection 
mechanisms 

Geo-Collaborative  crisis 
management 

NSF 

$  400  K 

M.  McNeese 
G.  Cai 

New  methods  and  techniques  for 
improved  crisis  management  via 
cognitive  aids  and  collaboration 
tools  for  teams  of  decision-makers 

Integrated  framework  for 
understanding  integrated 
intelligence 

SRI/NGA 

$  213  K 

D.  Hall 

T.  Shaw 

M.  McNeese 

J.  Wang 

Development  of  cognitive  aids  for 
automatically  processing  image 
and  non-image  intelligence  data  to 
support  a  “whole-brain”  analysis 
approach 

NTR:  Collaborative 
Research:  testing  and 
benchmarking 
methodologies  for  future 
network  security 
mechanisms 

NSF 

$  720  K 

P.  Liu 

D.  Miller 

G.  Kesidis 

Development  of  new  testing 
methodologies  for  network 

defense  mechanisms 

Analyst  Automation 

Task 

Contractor 

$  348  K 

J.  Yen 

M.  McNeese 
L.  Giles 

Development  of  computer  tools  to 
support  intelligence  analysts 
including;  multi-agent  framework 
for  analyst  collaboration;  adaptive 
search  engines;  problem-centered 
decomposition  and  analysis  aids 
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